Papers with end-to-end speech translation models
Fluent Translations from Disfluent Speech in End-to-End Speech Translation (N19-1)
Copied to clipboard
| Challenge: | Disfluency removal is an intermediate step between speech recognition and machine translation (MT) with the rise of end-to-end speech translation systems, disfluency recognition and removal needs to be incorporated into the model architectures or handled as a post-processing step. |
| Approach: | They propose to use a sequence-to-sequence model to translate from noisy, disfluent speech to fluent text with disfluencies removed using the recently collected ‘copy-edited’ references for the Fisher Spanish-English dataset. |
| Outcome: | The proposed model generates fluent translations from disfluent speech using the recently collected ‘copy-edited’ references for the Fisher Spanish-English dataset. |
CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech-to-text translation models can extract features from speech inputs, but they may include non-linguistic speech factors such as pitch, timbre and speaker identity. |
| Approach: | They propose a content-centric speech representation disentanglement learning framework for speech translation that decomposes speech representations into content representations and non-linguistic representations via representation disentanglement learning. |
| Outcome: | The proposed framework outperforms state-of-the-art speech translation models and cascaded models on five translation directions. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |